Skip to main content

View Post [edit]

Poster: MeditateOrDie Date: Jul 31, 2017 11:42am
Forum: faqs Subject: Re: robots.txt no longer supported to exclude page from being crawled?

A properly configured robots.txt will still prevent crawling -
I'm not aware of any change to that policy.

Note: Not all site indexers/crawlers will obey robots.txt
so if you don't want something accessible, configure
your servers to use permissions or other tricks to
prevent unauthorized access to places which
you'd prefer to protect.

Reply [edit]

Poster: hans.heiri Date: Aug 1, 2017 3:20am
Forum: faqs Subject: Re: robots.txt no longer supported to exclude page from being crawled?

Thanks for your quick reply! i gonna share this with our legal guys.

Thanks again,
hans